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Introduction 

In this brief, we examine how the ACT’s EPAS™ (Educational Planning and 
Assessment System) can be used to evaluate school-based intervention 
programs. Specifically, we consider the evaluation of the federal government’s 
Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR 
UP), an initiative designed to increase the college awareness and college 
readiness of students at low-income middle schools and high schools. 

What is EPAS? 

Composed of three integrated programs — EXPLORE®, PLAN®, and the ACT® 
test — EPAS is a longitudinal assessment system that measures student 
readiness in English, mathematics, reading, and science along a continuum of 
college readiness benchmarks. EPAS allows teachers, counselors, and students 
to track academic progress from eighth through twelfth grades on skills directly 
related and linked to college preparation. 

The knowledge and skills measured by the ACT are closely aligned with 
success in college. The higher a student’s score on the ACT, the more likely 
that student is to be college ready, to achieve higher grades in college, and to 
persist to a college degree. EXPLORE and PLAN are early measures of college 
readiness in the eighth and tenth grades, respectively, and provide scores that 
are reported on the same score scale as the ACT. 

In conjunction with assessing academic progress, EPAS helps students plan for 
their future by providing information on careers that match their interests and 
skills. Encouraging students to think about possible careers early on allows 
them to plan more effectively for the academic courses they will need to 
achieve their career goals. 

The GEAR UP Program 

GEAR UP provides discretionary grants to schools and partnerships for the 
purpose of increasing the readiness of low-income students to attend and 
succeed in postsecondary education. The grants are up to six years in length and 
provide services to a cohort of students who are then followed from middle 
school through high school. The services vary by school; some examples 
include tutoring and mentoring, programs to increase awareness of college 
admissions processes, programs to inform students about collegiate curriculums 
and relevant high school coursework, summer learning programs, and courses 
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for parents to improve their involvement and help with student planning. 
Because GEAR UP services vary across schools, we would expect the relevant 
outcomes to vary hy school also. For example, if the goal of the program is to 
educate students about the college admissions process, then meaningful 
outcomes might he taking college -preparatory courses, having college plans, 
and taking a standardized admissions test at the appropriate time. For programs 
that target specific academic skills (e.g., tutoring in math), achievement test 
scores may he the most appropriate outcome. A reasonable question to ask is 
whether a school or partnership can show that their GEAR UP program is 
having the desired effect. In other words, are there changes in the relevant 
outcomes that can be attributed to the GEAR UP program? 

EPAS as the Basis for Evaluation 

Evaluations of school-based intervention programs typically entail comparisons 
of schools (and students) who received the intervention to those who did not. A 
sound evaluation plan therefore requires collecting data on relevant outcomes 
and background characteristics of students and schools. To measure the effect 
of the intervention program, the data must be collected both before the program 
is initiated (baseline data) and during or after the intervention takes place 
(follow-up data). ACT’s EPAS is ideally suited for the evaluation of school- 
based intervention programs related to academic achievement, educational and 
career plans, and overall college readiness for the following reasons: 

• Baseline (EXPLORE) and follow-up (PLAN and/or ACT) measures are 
available so that changes in outcomes attributable to intervention 
programs can be assessed. 

• The EXPLORE, PLAN, and ACT assessments measure the same 
constructs at different time points, allowing for meaningful 
comparisons over time. 

• EPAS assessments include reliable measures of academic achievement 
directly related to college readiness standards. 

• EPAS data include important measures of students’ educational plans 
and vocational interests. 

• Data collected through EPAS include a wide variety of background 
factors such as parents’ educational level and race/ethnicity, which may 
be controlled for in analyses or used for analysis of specific groups 
within the population. 

• Evaluations can be performed on student-level data, on school-level 
data (by aggregating student-level data), or both. 

Evaluation of the GEAR UP Program 

ACT used EPAS data to evaluate the GEAR UP program. We compared 
changes in academic readiness and college intent for a sample of nearly 13,000 
students from more than 250 GEAR UP schools to a comparable sample of 
nearly 12,000 students from 250 non-GEAR UP schools. We used data from 
act’s explore and PLAN programs to measure students’ academic 
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readiness and eollege intent at grades 8 and 10, respeetively. In addition to 
eomparing GEAR UP and non-GEAR UP sehools directly, we also used a 
statistical model that controls for the poverty level of the schools. This was 
done because GEAR UP schools, hy definition, are schools with very high 
poverty levels. 

In these ways, we were able to measure the degree to which GEAR UP affects 
change between grades 8 and 10. Growth between these two grades is crucial 
for college readiness, because many students set their future educational 
courses during this period. Data from the ACT (grade 1 1 or 12) were not used, 
because most of the students we studied had not yet finished eleventh grade. 
However, in the future, the evaluation could be extended by also considering 
these data. Extending the evaluation to eleventh and twelfth grades would be 
especially meaningful because the data would then include students’ college 
preferences as well as final measures of academic achievement in high school. 

Results 

The results of the evaluation suggest that, compared to students from non- 
GEAR UP schools, students from GEAR UP schools perform slightly better 
with respect to changes in academic readiness and college intent from grade 8 
to grade 10. This is true also after controlling for poverty level. Specifically: 

• Students from GEAR UP schools had greater changes in overall 
academic performance from grade 8 to grade 10 (Table 1). 

Table 1: Changes in Mean Composite Score 



Mean Composite 
Score 


GEAR UP 


Non- 
GEAR UP 


EXPLORE (8th grade) 


14.53 


14.56 


PLAN (10th grade) 


16.36 


16.32 


Change 


+1.83 


+1.76 


GEAR UP increase 


+0.07 


After adjustment for 
poverty level 


+0.16 
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• Students from GEAR UP schools were more likely to be on target to be 
ready for college in English (Table 2) and reading (Table 3). 

Table 2: Changes in Percentages of Students Meeting English Benchmark 



Meeting English 
Benchmark 


GEAR UP 


Non- 
GEAR UP 


EXPLORE (8th grade) 


56.7% 


58.2% 


PLAN (10th grade) 


64.5% 


63.6% 


Change 


+7.8% 


+5.4% 


GEAR UP increase 


+2.4% 


After adjustment for 
poverty level 


+3.5% 



Table 3: Changes in Percentages of Students Meeting Reading Benchmark 



Meeting Reading 
Benchmark 


GEAR UP 


Non- 
GEAR UP 


EXPLORE (8th grade) 


32.5% 


34.3% 


PLAN (10th grade) 


36.6% 


35.1% 


Change 


+4.1% 


+0.8% 


GEAR-UP increase 


+3.3% 


After adjustment for 
poverty level 


+5.7% 



4 



• Students from GEAR UP schools were slightly more likely to take the 
college-preparatory high school core curriculum at grade 10 (Table 4) 
and have college plans at grade 10 (Table 5). 

Table 4: Difference in Percentages of Students Taking High School Core 

Curriculum 



Taking High School 
Core Curriculum 


GEAR UP 


Non- GEAR 
UP 


52.3% 


49.5% 


GEAR UP increase 


-1-2.8% 


After adjustment for 
poverty level 


-1-2.1% 



Table 5: Changes in Percentages of Students Having College Plans 



Having College Plans 


GEAR UP 


Non-GEAR 

UP 


EXPLORE (8th grade) 


89.1% 


89.1% 


PLAN (10th grade) 


87.9% 


86.6% 


Change 


-1.2% 


-2.5% 


GEAR UP increase 


-1-1.3% 


After adjustment for 
poverty level 


-1-1.2% 



Recommendations 

Based on our analysis, the following are some suggestions for evaluating 
GEAR UP programs: 

1. Tailor the analysis to the intervention. Eor example, if the goal of the 
program is to educate students about the college admissions process, then 
meaningful outcomes might be taking college prep courses, having college 
plans, and taking a standardized admissions test at the appropriate time. Eor 
programs that target specific academic skills (e.g., extra help with reading), 
achievement test scores may be the most appropriate outcome. Generally, the 
most appropriate outcomes will vary by GEAR UP program. 
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2. Follow students across time. This allows students and groups of students to 
show that they are indeed improving, and allows students to serve as their own 
baseline. 

3. Track students’ participation level in GEAR UP programs. 

Supplementing our analysis with data about the level of GEAR UP 
participation for each student, and the type of intervention each student 
received, will likely better isolate the effects of GEAR UP programs and make 
their evaluation more meaningful. 

4. Track students’ long-term outcomes. The true test of GEAR UP’s value 
occurs when students leave high school and have the opportunity to enroll in 
college. If possible, GEAR UP evaluators should track long-term outcomes, 
including college enrollment, retention, and degree completion, for the students 
that attended GEAR UP schools. 

5. Use a control group. Comparing outcomes for students from GEAR UP 
schools to a control group is an attractive study design, as long as the control 
group is similar with respect to the other factors that affect students’ college 
readiness. Possible control groups include: 

• The school itself. By comparing outcomes for students prior to the 
establishment of a GEAR UP program to those who come after, the GEAR 
UP effects can be measured. The strength of this approach is that school- 
level differences are naturally eliminated (as long as the school does not 
undergo extensive changes during the study period). Data must be collected 
for several years and care taken that GEAR UP effects are not confounded 
with other changes that may occur over time. 

• A similar school. By matching on a set of relevant variables, a similar 
school or schools can be selected for comparison. While this might be 
difficult for an individual school, ACT’s EPAS provides a rich source of 
data across thousands of schools. 

Conclusion 

With its reliable measures of students’ academic achievement, educational and 
career plans, vocational interests, and background characteristics, ACT’s EPAS 
helps program evaluators conduct powerful and meaningful analyses of 
changes in students’ readiness for college. EPAS is therefore a rich source of 
longitudinal data for the evaluation of school-based intervention programs. 
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